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and evaluation projects in adult literacy and basic education, pro- 
gram effectiveness, interagency coordination, and state assessment 
systems. She served as a member of the national advisory commit- 
tee for the Section 353 Adult Education Program Evaluation and is 
the author of numerous publications on assessment and evaluation 
of adult education programs, including Assessing the Ejects of 
Volunteer Literacy Programs (Laubach Literacy Action), Linking 
Research and Evaluation to Policy and Practice in Adult Educa- 
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Executive 

Summary 

This paper describes approaches that states have taken to systemic 
reform in adult education programs, approaches driven by the de- 
mand for accountability and high-quality services from legislation 
and other sources. Four areas are explored: 

• The use of innovative management, assessment, and instruc- 
tional practices 

• Development of policy that supports change 

• Professional and organizational learning 

• Coordination within and across states to leverage support for 
adult education 

The paper provides descriptions of state program improvement 
activities and discusses the conditions that influence adoption of 
change strategies. The importance of standards for learner 
performance, assessment instruments, data collection and use, and 
funding for performance is highlighted. 

Information on systemic change in adult education may be found 
in the ERIC database using the following descriptors: * Account- 
ability, Adoption (Ideas), * Adult Education, Change Agents, 
Coordination, Educational Innovation, Policy Formation, 
^Program Improvement, * State Agencies, State Programs, and the 
identifier ^Systemic Change. Asterisks indicate terms that are 
particularly relevant. 




Introduction 



During the 1990s, the field of adult basic and literacy education 
has faced the challenge of improving the quality of the services it 
provides to adult learners and the data about the results of these 
services. First noted in the Adult Education Act Amendments of 
1988 and in the National Literacy Act of 1991, and now empha- 
sized in the Adult Education and Family Literacy Act of 1993, this 
call tor accountability has focused the attention of policymakers at 
the federal, state, and local levels. The new legislation requires 
that states implement a system of program accountability, fund 
programs based, in part, on the quality of their services, and col- 
laborate with other agencies in reporting data on participant out- 
comes related to employment and further education and training. 
Central to carrying out these requirements is the states’ capacity to 
work with local adult education programs in providing training on 
data collection and use, in motivating programs to consider the 
adoption and development of new practices, and in promoting the 
role of adult basic education in the work force development and 
human services delivery system. 

Although many agencies in a state may have funds that can be 
used to support adult education activities, the key entity responsi- 
ble for leading a state's efforts in adult education is the agency that 
oversees the administration of funding from the Adult Education 
and Family Literacy Act. In the majority of states, this entity is the 
state's education agency. In 15 of the states. Pacific Islands, and 
territories, the community college system or a state agency other 
than K-12 education administers the federal adult education mon- 
ies. A state’s adult education office is responsible for distributing 
Adult Education and Family Literacy Act and state adult educa- 
tion funds, overseeing and monitoring local programs supported 
with these funds, and facilitating technical assistance and training. 
The adult education state office, and in particular, the state direc- 
tor of adult education, is often the primary spokesperson for adult 
education in state government and plays a kev leadership role in 
setting policy and promoting continuous program improvement. 

In response to the call for program accoiinrahilirv, data reporting, 
and high-quality services, state adult education offices have been 
engaged in various activities to develop performance standards, 
conduct program improvement initiatives, and align the deliverv of 
stall development with program needs. Some have undertaken 
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statewide initiatives to promote change throughout a state's adult 
education system, whereas others have carried out limited experi- 
ments in trying new ideas and processes. Both represent strategies 
that states can use to meet federal and state adult education fund' 
ing requirements and to heighten the quality of services that are 
available to eligible youth and adults (hereafter referred to as 
“adults”). As states have engaged in systemic change activities, 
they have both designed their own initiatives and used external 
consultants to assist them in program development. Increasingly, 
state staff have used information from the research, evaluation, 
and technical assistance adult education projects that are being 
funded by the U.S. Department of Education and the National 
Institute lor Literacy to carry' out these activities. 



Approaches in four areas 
that state policymakers 
can use in implementing 
systemic reform: 

• Stimulating the use of 
new management, as- 
sessment, and instruc- 
tional practices 

• Developing policy to 
support change 
strategies 



As state directors of adult education and their staffs have met na- 
tionally and regionally to discuss federal and state requirements, it 
has become evident that, although state adult education offices 
may he located in different organirational entities and have differ' 
ent local delivery systems, there are core issues that all states must 
address in developing continuous program improvement and ac- 
countability systems. These discussions among state and federal 
adult education officials have brought attention to the need for 
better dissemination and access to information, particularly with 
regard to the strategies that states are using to meet legislative re- 
quirements. As one step in the transfer of information to facilitate 
states' program development efforts, this document presents ap- 
proaches in four areas that state policymakers can use in imple- 
menting systemic reform: 

• Stimulating the use of now management, assessment, and 
instructional practices; 

• Developing policy to support change strategies; 

• Facilitating professional development; and 



• Facilitating professional 
development 

• Coordinating and collab- 
orating with agencies to 
leverage support for 
adult education services 



• Coordinating and collaborating with agencies to leverage sup- 
port for adult education services. 

The information presented in this document is drawn from the 
author’s work in documenting state program improvement initia- 
tives in California, Connecticut, and Pennsylvania as well as from 
work with other states and data sources. Although not compre- 
heiiMvc, the examples provided in this document are illustrative 
practices that state policymakers m adult education can undertake 
in refining their adult education <v stems. 
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Approaches to Systemic 
Change in Adult Education 



Defining Systemic Change 

In considering systemic change in the field of adult education, one 
needs to define what is meant by a “system” and what is meant by 
“change.” A system consists of organizations and the individuals 
who participate in these entities. In adult education, a system ope* 
rates at both the state and local levels and includes the following 
four components: 

• The state agency and staff who administer the federal and state 
adult education monies; 

* The local adult education program and staff who deliver ser- 
vices to adult learners; 

♦ The communities in which these services are located and the 
adult learners from these communities who participate in adult 
education services; and 

• The nonadult education state and local agencies and staff who 
provide support and other related services to adult learners. 

Although each component is discrete, the interaction and connec- 
tivity among the components constitutes a system. Any approach 
to systemic change must involve the organizations and individuals 
in each of these components. Inherent in this typology are the 
following assumptions: state and local adult education staff think 
that there is a need for change, they value working together to pro- 
vide high-quality services to adults, and they see state policy and 
funding as key levers for effecting change. There is a further as- 
sumption that nonadult education agencies and staff have critical 
roles to play in supporting the adult education services that learn- 
ers receive and in linking these learners to further education and 
training. The expected result from these efforts is a high-qualit v 
system ot adult education services that enables adult learners to 
achieve their educational goals and improve the quality of their 
lives in their roles as family members, workers, and citizens. 
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State and local programs 
must work together in try- 
ing new processes and ac- 
tivities, such as assess- 
ment, evaluation, new cur- 
ricular content, and pro- 
gram regionalization, in 
order for the system to 
work more effectively in 
serving adult learners. 



Change must occur across components in order for it ro he consb 
dered systemic. In states undertaking systemic change initiatives in 
adult education, state staff have used policy development, staff 
training, and technical assistance as levers for promoting change 
activities. The assumption has been that state and local programs 
must work together in trying new processes and activities, such as 
assessment, evaluation, new curricular content, and program 
regionalization, in order for the system to work more effective! in 
serving adult learners. 
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Framework for Understanding Change 

Recent work on systemic reform in elementary and secondary edu- 
cation, particularly in mathematics and science education, provides 
a context for examining current efforts at sysrems change in adult 
education. Many of the elementary and secondary education ini- 
tiatives have drawn on the research on change conducted over the 
past 30 years in order to understand the conditions under which 
organizations and individuals are likely ro engage in new practices 
(e.g., Crandall and Associates 1982: Gross. Giaequinta. and Bern- 
stein 1971; Hall and Loucks 1978; Rogers 1%2). In a review of 
systemic reforms and the mathematics and science classroom, 
Knapp (1997) provides a lens through which to understand three 
kev element'' of reform: 



• The role of innovation in effecting change 

• The role of police implementation 



• The role of professional and organizational learning 

Each ol these elements is based on a set of assumptions and condi- 
tions under which change mav he possible. The elements are inter- 
related and mav he considered to he critical components of an ini- 
tiative to promote >\>iemic change. In particular, they provide a 
context for examining the strategies and incentives that state adult 
education staff arc tome in encouraging local adult education pro- 
gram stalf to develop new approaches to organizing services and 

meeting the educational needs of adult learners. 
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The Role of Innovation 

An approach to improving programs in the field of education that 
began in the early 1970s was to train teachers to adopt innovative 
curricula, instructional strategies, or assessment practices that had 
been shown to be effective in facilitating student learning or attain- 
ment of goals. The assumption underlying this approach was that 
training instructors to use new, effective practices would lend to a 
higher quality of service that, in turn, would increase the academic 
performance of learners. One way in which this approach was im- 
plemented was with funding provided by the U.S. Department of 
Education's (formerly the U.S. Office of Education) National Dif- 
fusion Network (NDN). Through the NDN grants, program de- 
velopers in local school districts, nonprofit organizations, and other 
educational entities disseminated their practices nationally by 
training teachers and other school personnel and providing techni- 
cal assistance in the use of these practices. The intent was to re- 
duce the time instructors spent attempting to solve similar prob- 
lems by leveraging the use of innovative practices that had been 
developed with federal program and discretionary funds and that 
had evidence of effectiveness through a rational dissemination 
system. Although the majority of NDN-funded programs were in 
elementary and secondary education, five adult education pro- 
grams were nationally disseminated through the NDN (Moore 
1996). The NDN operated for over 20 years and was the most 
comprehensive federally supported dissemination system in educa- 
tion. In a more limited way, innovative practices have also been 
promoted through state dissemination systems and with funding 
from foundations. 

The use of formal dissemination systems mav be one wav to trans- 
fer innovative practices. Another option is for states to encourage 
administrators and staff to try new practices as a way of improving 
their programs. This can he done through state initiatives that 
motivate programs to try new approaches to program services, 
through mandates that require local programs to use specific pro- 
cesses or that encourage change in local programs intormalh . Re- 
gardless of the way in which individuals learn about new practices 
or are encouraged to use them, it appears that considering program 
alternatives is one component of systemic change. 

This strategy of using innovative practice* to effect change in edu- 
cational organizations R based on a number of assumptions: 
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• Practitioner* and policvmakers believe that programs need to 
be improved and that the adoption oj effective practices is one 
way to address this need. 



• There is a practice, product, process, or organizational arrange- 
ment that has been shown to be elective and that can be dif- 
fused, disseminated, or adopted. 

• There are fiscal resources to support the dissemination and 
adoption processes. 

• Change agents are critical to the dissemination and adoption 
processes. 



Stimulants for Change 

in the field of adult education, the impetus for program improve- 
ment has come from many sources. Sometimes state leaders deter- 
mine that a new initiative is needed to focus the attention of local 
programs on issues such as serving new target populations, provid- 
ing better services, or raising awareness about the importance of 
adult education and literacy within and outside of education. Of- 
ten a state or federal directive may prompt a state adult education 
office to implement a statewide program improvement initiative. 
Changes in program services can come from local program staff 
who identify a need for new services and suggest ways of addressing 
this need. Program staff also may participate in action research or 
practitioner inquiry activities to collect data about their practices 
and examine them in light of recent research. 

State-Driven Initiatives. California’s competency-based 
adult education (CBAE) initiative that began in 1982 was an ex- 
ample of* a state-initiated mandate whereby the state’s federally 
funded adult education agencies were required to use a compe- 
tency-based approach in organizing and delivering services to 
learners. The state adult education staff viewed the mandate as a 
mechanism lor facilitating program coordination and for increasing 
the quality of adult basic education services being offered to adults. 
Although agencies were expected to organize their programs to in- 
clude kev components of a competency-based system, such as spe- 
cification of competencies, procedures for placement, coordinated 
instruction and a^csMVient, use of appropriate materials, and pro- 
cesses tor reporting student performance, thev had flexibilitv in 
specifying competencies and program content. The expected re- 
sult from the CTv\E mandate wa^ better procedures for program 
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accountability as well as a common reference tor communicating 
and sharing information that would result in the overall improve- 
ment of adult basic education in California (Alamprese er al. 

1987). 

The Connecticut Department of Education’s Bureau of Adult Edu- 
cation’s (now the Bureau of Adult Education and Training) 5 wear 
initiative — the Connecticut Adult Performance Program 
(CAPP) — is an example of a state initiative promoted by a state 
directive. The bureau began CAPP in response to a recommenda- 
tion made by the state’s Adult Education Study Committee and 
endorsed by the State Board of Education in 1985, which stated 
that all basic skills programs adopt a competency-based approach 
to delivering services by 1990 as a condition for funding from the 
Connecticut Department of Education. CAPP was expected to 
meet three needs regarding adult education services in the state: 

1. To have a comprehensive adult basic skills system that inte- 
grates academic skills with functional life skills 

2. To establish a program management system that provides in- 
formation that can he used ro assess program accountability 
and to make funding decisions 

3. To increase the performance of adult learners participating in 
state-funded ba'sic skills programs 

The developers of the CAPP initiative assumed that staff develop- 
ment and technical assistance activities would be effective mech- 
anisms for disseminating new policies and programmatic interven- 
tions to adult education administrative, instructional, and counsel- 
ing staff. It was also expected that these policies and programmatic 
interventions would result in more relevant adult education in- 
struction and accountable adult education programs (Alamprese 
1993). 

Although the California and Connecticut statewide improvement 
initiatives predated the accountability requirements in the Adult 
Education Act, they both accurately predicted the need for more 
accountability and better services in adult basic education. These 
initiatives were also systemic in that they dealt with all levels of the 
adult education system, and they included policy, development, 
training, and technical assistance. 



A more recent state effort to footer systems change is Pennsylvan- 
ia's multiyear program improvement initiative. Project Educational 
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a 

State and local program 
staff desiring to improve 
their systems or programs 
need to access existing 
effective practices or 
develop new ones as part 
of the change process. 



Quality in Adult Literacy (EQuAL), which was begun in 1994 
partly in response to the federal call for accountability and also due 
to the state adult education office’s desire to foster an improved 
system for adult education in the state. The Bureau of Adult Basic 
and Literacy Education began EQuAL as a process for collecting 
and analyzing learner and program data that the bureau could use 
in monitoring the performance of state-funded adult education 
services, determining performance standards for local adult educa- 
tion programs, and identifying ongoing staff and program develop- 
ment needs. 

The state’s process for monitoring adult basic education and liter- 
acy programs also was revised. Over time, EQuAL has become the 
focus of program improvement efforts in the state that has involved 
the alignment of staff development, technical assistance, and state 
policies supporting these activities (Alamprese, Voight, and 
Stickney 1998). 

Local Change Efforts. In addition to state-driven improve- 
ment efforts, local program staff have undertaken change activities 
hv initiating action research (e.g., California’s OAR project) and 
practitioner inquiry activities (Lytle 1996). Local adult education 
staff work together to pose questions about the practice of adult 
basic education and literacy or to examine various aspects of their 
programs. They collect information and examine this information 
in light of research findings or practices of others. The staff meet 
to discuss their research findings and their implications for under- 
taking new practices. This approach to change is locally driven 
and is not necessarily undertaken in direct response to a state 
initiative. 

The Use of inno vative Practices 

Regardless of the impetus for change, state and local program staff 
desiring to improve their systems or programs need to access exist- 
ing effective practices or develop new ones as part of the change 
process. In the field of adult education, the Special Experimental 
Demonstration projects supported under Section 353 (formerly 309 
and 310) of the Adult Education Act (as amended by the National 
Literacy Act of 1991) have been the main sources of promising 
practices. With these funds, special project staff have developed 
materials for curriculum and instruction, designed approaches for 
delivering services, conducted research and evaluation, developed 
assessment systems, and disseminated information. Since the 
R370s, several thousand projects have been supported with these 
monies, ranging from large, multiyear projects directed at systems 
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change such as the External High School Diploma Program (EDP), 
the Comprehensive Adult Student Assessment System, and the 
ESL Institute, to smaller, 1-year efforts in materials development 
and dissemination, among others. Four of the five adult education 
projects funded under the National Diffusion Network were 
developed with these monies: Adult Performance Level (APL) 
Program, BES Adult Literacy Project, Comprehensive Adult 
Student Assessment System, and the EDP. 

Although numerous projects have been supported with the Special 
Experimental Demonstration funds, the national evaluation of the 
Section 353 Set-Aside for Teacher Training and Innovation in 
Adult Education found that a limited number had been evaluated 
and disseminated (Mackin et al. 1996). Examined in this study 
was a sample of 116 projects (from a universe of 1376) carried out 
during the period Fiscal Year 1991-Fiscal Year 1994. The evalua- 
tion addressed a number of issues concerning the extent to which 
the projects could be viewed as effective and useful in improving 
practice. For example, few of the projects had a required evalua- 
tion component or data describing their effectiveness. Most often, 
it was not clear whether the project had been developed based on 
prior research or existing knowledge. Although there was an intent 
to disseminate the projects, there was no system for transfer that 
included personnel and resources to facilitate the adoption or 
adaptation of these practices. 

The evaluation offered a number of recommendations for state 
adult education officials that are important to consider in examin- 
ing the role of innovation and effective practice as part of the sys- 
temic change process in adult education. These suggestions are 
particularly important in light of the State Leadership monies that 
replace Section 353 funds in the Adult Education and Family Lit- 
eracy Act of 1998. To ensure the likelihood that special projects 
developed with State Leadership funds can be of high quality and 
useful in effecting change, the following should be considered: 

• Include sufficient time and resources in tunding a project to 
determine the types of research and prior experience that can 
he used in formulating or conceptualizing the project. 

• Require that projects use the stages of development-demon- 
stration-revision in their design to ensure that a high-qualitv 
product is created. 

• Require that projects include plan'* tor evaluation and dissemi- 
nation from the outset. 
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• Fund projects that interrelate or complement each other so 
that they can he used in building the components of a systemic 
change process. 

Pennsylvania’s activities in funding Section 353 special projects to 
support Project EQuAL are one illustration of the use of these rec- 
ommendations. Central to the Bureau of Adult Basic and Literacy 
Education’s development of a performance management system 
has been a reliable and valid assessment system. One of EQuAL’s 
first activities was to review the quality of the learner assessment 
data being collected by local programs. Based on the results of this 
review, the bureau determined that program staff needed both in- 
formation and training regarding the types and uses of assessment, 
and thus supported a Section 353 project to produce an assessment 
manual and related training. Another element of the management 
system was having a set of competencies to guide the development 
of program content. To address this need, the bureau funded a 
project to review prior research and development in the specifica- 
tion of life-skill competencies for adult education and then to 
adapt a life-skill competency framework with the underlying basic 
skills that Pennsylvania’s local adult education programs could use. 
To support these activities, the bureau included training related to 
assessment and competency use as part of the state’s regional staff 
development system. In this way, multiple Section 353 projects 
were funded as part of an integrated approach to creating and 
implementing a performance management system for adult basic 
education and literacy services. 

Conditions Supporting Adoption of Practices 

Funding program staff to develop new practices is one method of 
facilitating program change. Encouraging them to adopt or adapt 
effective practices designed by other adult educators is another 
strategy. As previously noted, the United States currently does not 
have a national dissemination mechanism to facilitate the adoption 
process. However, adult educators do disseminate their practices 
through staff development events, conference presentations, the 
use of 'online systems and education clearinghouses, and the paper 
transfer of information. These dissemination activities are an ini- 
lial step in promoting the adoption or adaptation of new practices 
and can be effective in discouraging adult educators from develop- 
ing redundant materials, curricula, other tools that they use in 
working with adult learners. 

Inherently appealing in reducing development time and resources, 
the adoption of new practices can "till he a complicated process. 
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The findings from Knapp’s (1997) review of the role of innovation 
in systems change is helpful for understanding the conditions under 
which the adoption of a new practice is likely to he successful. 

Four conditions appear ro he important: 

• Cost is reasonable. 

• Innovation is not too complex. 

• New practice is congruent with prior practices and users’ 
knowledge. 



• Change agents understand the state of the existing program 
and its readiness for change. 

Cost and Complexity. The cost of the new practice is a criti- 
cal element, and its reasonableness is a relative term that will vary 
according to the resources available in a local adult education pro- 
gram. Related to cost is the complexity of the innovation or prac- 
tice. For example, a multifaceted assessment process, such as the 
External High School Diploma Program, is more complex to adopt 
than curriculum materials used in teaching writing for the General 
Educational Development (GED) Test. The history of the dissemi- 
nation of the External High School Diploma Program through the 
National Diffusion Network and subsequently through the Ameri- 
can Council on Education is a case example of an innovation in 
adult education that has been both expensive and complex, and 
thus has been limited in its adoption. As originally designed, the 
EDP required multiple stall to he trained over 3-5 days, which was 
modified over time. The implementation of the program also call- 
ed for staff to conduct one-on-one assessment with adult learners, 
which is more costly than carrying out assessment in a group set- 
ting (Alamprese and Feldman 1980). One result of these program 
elements has been the reluctance of stale and local adult education 
programs to invest in this alternative secondarv credentialing pro- 
gram, even though it is an appropriate option for adults with life 
experience from working or managing a lamily. 

A contrasting example has been the dissemination of the Compre- 
hensive Adult Student Assessment System. Although a difterent 
type of assessment process from the EDP, CASAS also was dissem- 
inated through the National Diffusion Network and continues to 
he available nationally. Some factors that facilitated the adoption 
of CASAS include a shorter training requirement, a les<-costlv 
implementation, and options for using different products developed 
hv the system (Educational Programs that Work 1995). 
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Having sufficient resources 
available for staff to re- 
ceive training and use a 
new practice is critical. 



The experiences of the dissemination ot these two adult education 
programs indicate that having sufficient resources available tor staff 
to receive training and use a new practice is critical. The more 
complex a practice (i.e M a practice with multiple components, 
highly technical information), the more time and fiscal resources 
will be needed for its successful adoption. 



f f Congruence of Practice. In addition to cost and complexity 
ot a new practice, the condition of the congruence of the new prac- 
tice with prior practices and the knowledge of the intended user 
influence adoption. For example, both the EDP and CASAS are 
competency-based processes. In the initial dissemination of these 
programs, the early adopters were more likely to be states and 
programs that were using a competency-based framework that al- 
lowed the integration of these programs philosophically into cur- 
rent practice. This philosophical congruence made it more likely 
that staff would accept a new practice and use it. 



Role of Change Agent. A fourth element is the role of a 
change agent and this individual’s understanding of the extent to 
which a program is ready for change. The change agent can be a 
kc . person in a program’s staff such as the program coordinator or 
resource teacher, who works with the staff in facilitating the adop- 
tion and use of the new practice. A key role of the change agent is 
to assess the extent to which the practice being considered for 
adoption is similar philosophically to the staffs existing beliefs and 
behaviors. It a philosophical change on the part of the staff is an 
element ot the improvement process, staff may need additional 
time, training, and materials to accept the new perspective and in- 
corporate it into their practice. The replication of a practice us- 
ually requires that staff use the practice in the way it was intended 
t o obtain the same results. If staff do not have the same success 
using a new practice, then it may he helpful to review the assump- 
tions the staff made about the use of the practice or the way in 
which it is being implemented, ll the programmatic or client con- 
ditions of the adopter arc different from those ot the developer ot 
the practice, then the practice may not he a good fit with the pro- 
gram attempting ro use it. 



All of these factor* are important in fostering change through the 
use ot new practices developed bv others. Where possible, state 
policymakers and local program staff should seek evidence that the 
practice has worked under similar conditions and understand the 
tvpes of outcomes that are reasonable to expect from the use of the 
practice. One strategy that state policymakers have used is to im- 
plement a new initiative in stages that involve supporting pilot 
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programs who try out new practices under different sets of condi- 
tions. By selecting pilot programs that reflect the variety of service 
providers and clients, state policymakers and local staff can deter- 
mine the conditions under which a new practice is likely to pro- 
duce the expected outcomes. This information will heip to guide 
program staff in integrating new practices into existing operations. 



The Role of Policy 

Any systemic change initiative must have policies that guide the 
reform activities. The assumption is that a policy, a publicly an- 
nounced set of intentions, combined with an allocation of re- 
sources and other provisions, will motivate and sustain change- 
oriented activities throughout a system (Knapp 1997). Since the 
passage of the 1988 Adult Education Act Amendments and the 
National Literacy Act of 1991, state offices of adult education have 
been concerned with policy development in the area of program 
accountability, particularly concerning program quality and evalua- 
tion. Much of this work has centered on the specification and im- 
plementation of indicators of program quality. A review of the 
1992 state adult education plans conducted by Elliott and Hayward 
(1994) indicated that all states had adopred the required indicators 
of recruitment, retention, and learner progress. All but four states 
at this time had specified some measure of learner advancement 
(e.g., advancement to a higher program skill level, attainment of a 
GED or competency certificate, and entry into other programs). 

With the passage of the Adult Education and Familv Literacy Act 
of 1998, states now have to set policy for adult education programs 
regarding performance levels in three areas. The core indicators of 
performance for these three areas are — 

• Demonstrated improvements in literacy skill levels in reading, 
writing, and speaking the English language; numeraev and 
problem solving; English-language acquisition; and other liter- 
acy skills 



• Placement in, retention, or completion of poscseeondary 
education, training, unsubsidiied employment, or career 
advancement 

• Receipt of a secondary school diploma or its recognized 
equivalent 
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In addition, states need to describe how their adult education and 
family literacy activities will he integrated with other adult educa- 
tion, career development, and employment and training activities 
in the state or outlying area served by the agency eligible to receive 
funds. 

Because of* the requirements of the 1998 legislation, states arc de- 
veloping policy directives for guiding local adult education pro- 
grams in conducting learner assessment and other data collection 
and reporting results to the state agency. The topic of assessment 
is a key policy area for adult education and a linchpin to the sys- 
temic change activities underway in the states. State adult educa- 
tion offices also have specified policies in related areas, such as pro- 
fessional development, to encourage local programs to carry out 
continuous program improvement activities. 



Policy and Program Accountability 

Challenges to Accountability 

The impediments to state adult education efforts to enhance pro- 
gram accountability and program evaluation have been well noted. 
Recent studies (Kutner, Webb, and Matheson 1996; Moore and 
Stavrianos 1995; Young et al. 1994) have suggested a number of 
factors that have inhibited local programs 1 capacity to implement 
accountability processes: 

• The different types of curricula used by local adult education 
programs have made it difficult to align assessment and 
instruction. 

• The administration of standardized pro- arid posttests has not 
been a priority in adult education programs and has affected 
the frequency and quality of assessment data collected. 

• Assessment instruments often are selected tor ease of adminis- 
tration rather than because they reflect the content of what is 
being taught. 

Another factor has been the dilticultv in gathering and interpreting 
data accurately at the Mate and local leveh. For example, pro- 
gram.'- that have open-entry and open-exit policies whereby learn- 
er' can enter or complete a program at anv time often have 
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short-term attendance that makes it difficult tor program adminis- 
trators to track how many adults have been served or the types of 
services that they have received from a program. According to 
estimates made by the National Evaluation of Adult Education 
Programs (Young et al. 1994), errors in local administration of 
standardized tests invalidated as much as one-third of the perfor- 
mance data collected as part of the study. 

Attempts to address these challenges have come from the federal 
and state levels. Since 1991, the U.S. Department ot Education’s 
Office of Vocational and Adult Education has supported state ef- 
forts to improve accountability through the development of an 
evaluation framework, the specification of quality indicators for 
adult education, and the funding of technical assistance for state 
accountability and assessment (Condelli and Kutner 1997; 

Condelli, Koloski, and Webb 1992; Kutner, Webb, and Herman 
1993). The National Institute tor Literacy’s project to develop 
accountability systems and its ongoing Equipped for the Future 
initiative to specify content standards for adult basic education 
with related assessment are two other supports for the account- 
ability movement in adult education (National Institute for Lit- 
eracy 1995; Stein 1995). 

Some states have exhibited leadership in designing more account- 
able adult education programs. These activities have included 
developing consensus about what should be taught, designing a 
core adult education curriculum that often emphasizes employ- 
ment-related basic skills, and identifying assessment instruments to 
measure what has been learned by participants. 

A 1996 review of 1 1 state efforts to develop a statewide system of 
learner competencies, curriculum, and assessment indicated that 
seven states had specified statewide competency systems, with four 
of the seven states drawing their competencies from CASAS. 

Some of the states had produced statewide curriculum guides with 
recommended resources, whereas others had suggested curriculum 
that local programs might consider. A tew of the states also devel- 
oped statewide procedures for recognizing learners’ accomplish- 
ments through certification programs (Kutner, Webb, and 
Matheson 1996). 

Key Issues in State Accountability 

As states move ahead to address the accountability requirements in 
adult education in the context ot program improvement, the 
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development of state policy plays a key role in these activities. 

Four areas in which policy is an important lever are the following: 

1. Setting standards for learner performance 

2. Using assessment instruments 

3. Promoting the collection and use of high-quality data 

4. Funding programs based on performance 

Setting Standards. States face a challenge in attempting to 
set standards for learner performance and having sufficient data to 
specify the level ot gain learners are expected to achieve. One type 
of standard is a relative standard, in which the expected level of 
performance is set based on the actual average performance of 
learners in a state’s system. In order to set a relative standard, a 
state must have reasonable data about the performance of adult 
learners in various levels of a program (e.g., beginning, intermedi- 
ate, advanced). The average gain of learners over a specified 
period of time (the number of hours of instruction a learner re- 
ceives) is computed for each level of learners. These performance 
levels can be validated as the quality of learner assessment data 
increases over time. 

Both Connecticut and Pennsylvania used this methodology in set- 
ting their performance standards for adult education. In Connecti- 
cut, the state’s data on adult basic education learners participating 
in the Job Opportunities and Basic Skills Program GOBS) were 
used to set the standards for determining the satisfactory progress 
of these clients in education programs. The analysis was expanded 
to include all adult basic education participants and standards then 
were set for the state's adult basic education system (Alnmprese 
1998). In Pennsylvania, the external consultant to Project EQuAL 
collected and analyzed assessment data from the project’s 20 pilot 
sites and developed draft performance standards that were then 
validated based on the analysis of data from a larger number of 
programs. 

Another type uf standard is an u/\so/me standard, which is based on 
the level of performance that it takes a learner to achieve a certain 
outcome (e.g., obtain a job, advance in a job, obtain a high school 
diploma). This* type ot standard is directly related to the expected 
outcome^ from program participation and, as such, would he a 
more accurate standard. However, since little national or state 
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data exist for adult education that can relate the amount of gain a 
learner achieves to specitic outcomes, this type of standard is rarely 
used. 

Using Assessment Instruments. As part of the standards* 
setting process* states are having to specify a policy on the types of 
learner assessment instruments that local programs use to measure 
learner progress. Some states (e.g., Connecticut* Oregon) require 
that local programs use specitic instruments in assessing adult basic 
education and English-as-a-second-language learners. Other states 
(e.g., Pennsylvania) give programs flexibility in choosing from a list 
of instruments. What is important in determining this policy is the 
appropriateness of the instrument for the types of learners and pro- 
grams and the ways in which data will be analysed and used. 

Discussions at the National Forum on Adult Education and Liter- 
acy: Views from Teachers sponsored by the U.S. Department of 
Education in September 1998 reinforced the need for the dissemi- 
nation of information to local program staff about the types and 
uses of learner assessment instruments. For example, assessment 
data can be used to — 

• Place learners in a specitic level of instruction; 

• Measure a learner’s progress on an ongoing basis relative to 
his/her own performance; and 

• Measure a learner’s progress from one time period to another 
compared to other learners. 

The type of instrument one chooses (e.g., standardized, nonstand* 
ardized; general literacy measure; specitic skill assessment) should 
depend on the intended uses of the assessment. For example, a 
diagnostic or placement instrument (e.g., CASAS Placement Test, 

TARE [Test of Adult Basic Education] Locator) is used to place 
learners in an appropriate level of instruction or to guide instruc- 
tion for learners in a multilevel class. This type of instrument con- 
tains fewer items and is administered in a shorter amount of time 
than is an instrument used to measure learners’ progress. Because 
a placement or diagnostic instrument mav not assess a wide range 
of skills it should not be used for measuring progress; rather, a stir- 
vcv achievement test is the appropriate instrument to determine 
learner gain. 

A related issue concerns the use of measures of generalized literacy 
TARE. ABLE, CASAS, TALS |Tos of Applied 
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Literacy Skills] — with beginning learners who have reading diffi- 
culties. The generalised measures do not necessarily identify the 
specific reading skills that learners will need to develop, and spe- 
cific reading instruments may be needed to supplement the mea- 
sures of generalized literacy skills. 



a 



What is critical is that all 
program staff understand 
the variety of assessment 
instruments that are avail- 
able and their appropriate 
use. 
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There has been increasing interest in the adult education field in 
alternative or applied assessment that enables learners to present a 
variety of evidence of skill demonstration. Many instructors cur- 
rently use alternative assessment to monitor learners 1 progress in 
achieving the goals thar they have set. Although alternative as- 
sessment is appropriate for documenting individual learners’ ac- 
complishments, it cannot be used for comparing learners 1 progress 
unless the assessment is standardized. In this circumstance, the 
same items are evaluated under the same conditions with all learn- 
ers. What is critical in the assessment discussion is that all pro- 
gram staff understand the variety of assessment instruments that 
arc available and their appropriate use. 



The determination of the types of assessment instruments that 
states will use in meeting their accountability requirements is a 
critical policy decision. The challenge is for state policymakers to 
collect learner data that can be aggregated across programs while 
considerating the strengths and weaknesses of the available instru- 
ments, the variety of learners being served by programs, and the 
state's capacity to provide ongoing training and technical assist- 
ance to assure the reliable and valid collection of assessment data. 



Promoting the Quality and Use of Data. The quality of 
standards a state can set is dependent on the available data. As 
the experience of states has indicated (e.g., the training and tech- 
nical assistance provided in Connecticut, Pennsylvania, and Ore- 
gon), local programs need extensive training and follow-up in the 
types of appropriate assessment, the administration of instruments, 
and the interpretation of results. One type of policy states have set 
ro increase the qualitv of data is to require that all instructors he 
trained in methods of assessment administration. Connecticut's 
CAPP initiative (and the subsequent Connecticut Competency 
System) is an example of how new adult basic education instruc- 
tors are expected to receive training in the administration and use 
of the CASAS instruments as part of their orientation to a pro- 
gram. Through Pennsylvania’s Project HQuAL, assessment train- 
ing is being made available ro local adult basic education staff’. 
Since the beginning ot the competenev-based initiative in Califor- 
nia, assessment training has been an integral pan ot ihc "late’s Malt 
development program. 
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A kev factor in promoting high-quality data is the training of local 
program staff in the multiple purposes and uses of data. Often 
program staff view learner assessment and program data as infor- 
mation that must he submitted to the state lor accountability pur- 
poses. In fact, the primary client for these data is the learner, fol- 
lowed by the instructor and the program administrator. Assessment 
and related program intake data about a learner can provide the 
following types of information that instructors can use in guiding 
instruction: 

• (deals for participating in a program 

• Learning strategies and the role of literacy in learners’ daily 
lives 

• Artitudinal disposition toward learning 

• Skills and knowledge at time of entry into a program 

• Progress toward skill and goal attainment 

Instructors can use data about learners’ goals for participation and 
skills and knowledge at time of entry into a program to develop an 
individualized learning plan that is reasonable given the learners’ 
goals, prior experience, and skills. Providing ongoing assessment 
information to learners can enable them to monitor their own pro- 
gress and identity the skill areas in which they may wish to focus 
their study. This information also helps learners to identify their 
strengths and reinforces their participation in a program. 



As the second beneficiary of data, instructors can use assessment 
information to target instruction so that learners receive adequate 
assistance in specific areas of need. Assessment results also provide 
a picture of the overall performance of a class and may signal t hat 
changes are needed in instructional methods or materials. Further- 
more, this information can he helpful in indicating that learners 
have needs that are impeding their progress. 

There are multiple wavs in which an administrator can use data 
effectively in managing a program. An examination of assessment 
results by class and level can indicate the overall strengths and 
weaknesses of a program as well as needed changes in types and 
levels of instruction. This may in turn indicate that staff need 
training in particular content or instructional methods. Variable 
performance by individual learners (in contract to iho performance 
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ot a whole class) can signal that learners may require services in 
addition to chew ottered by the adult education program. 

At the program level, administrators can use assessment and demo- 
graphic data to determine the populations of adult learners not be- 
ing served and the extent to which a program meets the state’s per- 
formance standards. This information also can be used in awarding 
learners certificates of progress or credentials. For example, states 
are developing learner certificate programs that will be used to doc- 
ument and recognize learners’ progress from one level to the next 
in an ABE or English-as-a-second-language program. Part of this 
documentation may include the competencies that a learner is ex- 
pected to demonstrate at each level of proficiency. 

If viewed from the perspectives of the learner, instructor, and ad- 
ministrator, assessment and program data are valuable tools in 
guiding learning, facilitating the delivery of services, and managing 
a program. The more value that a program places on the data that 
it collects, the higher in quality the data are likely to be. 

Funding Programs Based on Performance. A third 
emerging policy area concerning assessment is the funding of pro- 
grams based on their performance. States are beginning to con- 
sider the relationship of implementing standards to funding. This 
is a complicated issue with many facets. For example, one consid- 
eration is the extent to which all programs, regardless of funding 
level, will be held to the same levels of performance. Related to 
this is the relationship of equity to performance funding. As states 
begin to develop practices related to performance funding, they 
will have to consider the types of allowances that might be made 
when unexpected events, such as an influx of the target population 
of learners in a community, impedes a program’s capacity to meet 
its performance goads. One approach might he to consider a range 
of factors concerning performance (e.g., learner, attendance, per- 
cent ot target population served) that would be used to assess 
whether a program has met its performance objectives. 



The Role of Professional and Organizational Learning 

A key element in any systemic change activity is the role of devel- 
opment ot both professionals and the organizations in which these 
individuals work. The underlying assumption is that instructors 
involved in reform mav not possess all of the necessary knowledge, 
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skills, and attitudes that are needed to cam' out the reform, so they 
may need to engage in a long-term process of learning. A related 
view is that lifelong learning is a necessary component of all work, 
since the demands required for work are likely to change over time. 

In the field of adult education, this change process is known as 
continuous improvement. Another assumption that often is not 
acknowledged is that individuals at other levels of the system who 
support instructors’ work (e.g., administrators) have a similar learn- 
ing challenge. Furthermore, organisations possess qualities that are 
greater than the aggregate of individuals within it, and there is 
learning that can be associated with the organization as a whole 
(Knapp 1997). 

Almost all of the states support professional development for local 
adult education staff to some degree (Mackin et al. 1996). The 
types and extent of these activities vary considerably, with short- 
term workshops and conference participation still a major part of 
the staff development system in adult education. Some states have 
centralized the delivery of professional development as a way of 
coordinating training opportunities, disseminating practices, and 
encouraging ongoing improvement activities. 

More recently, attention is being paid to the administrators who 
lead the local adult education programs. Any systemic change ac- 
tivity must include these individuals in the policy discussion and in 
transferring information concerning new practices or requirements 
to instructional and other staff. Many states regularly convene 
their local program directors to discuss regulatory and reporting 
issues. Increasingly, these discussions have included information 
about the setting of standards, the collection of assessment data, 
and the availability of professional development opportunities tor 
staff (e.g., program director meetings held in Oregon and Alaska). 
As the link between the state adult education and local program 
staff, the administrator is the interpreter of state policy and has a 
key role to play in providing leadership in change activities 
(Quigley 1997). 

The third level of learning occurs in an organization. The rules 
guiding an organization may need to he adapted to accommodate 
new practices and policies. As noted in the previous discussion on 
the adoption of innovation, the culture of an organization and its 
readiness to accept change arc important factors in any program 
improvement process. Administrators and other leaders (e.g., pro- 
gram coordinators, resource teachers) in an adult education pro- 
gram have the responsibility of setting a vision, involving stnfl in 



u 

The administrator is the 
interpreter of state policy 
and has a key role to play 
in providing leadership in 
change activities. 



a 

The culture of an organi- 
zation and its readiness to 
accept change are impor- 
tant factors in any pro- 
gram improvement pro- 
cess. 

ff 



29 



21 



Approaches to Systemic Change 



determining the steps that can he taken to implement the vision, 
and creating a community of learning so that each new practice or 
policy undertaken as part of a program’s improvement process is 
reviewed and assessed. In this way, all participants have a stake in 
the outcomes and a voice in the way the outcomes are achieved. 



i 
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Leveraging Support 
through Coordination 
and Collaboration 

Within-State Activities 

The final elements in systemic change are the activities that state 
adult education offices need to undertake in working with partners 
within education, across agencies serving the same subpopulations 
as their agencies, and with other states. Within a state, the adult 
education office can work with other offices in education serving 
adult learners as well as with other agencies (e.g., Labor, Human/ 

Social Services, Corrections, Higher Education) and public/private 
organizations serving similar subpopulations in conducting the fol- 
lowing activities: 

• Developing/using joint data system with common performance 
outcomes 

• Conducting/supporting joint data analysis 

• Conducting joint staff development 

• Conducting joint projects with state volunteer organizations 

In order for these activities to occur, state adult education staff 
must establish a working relationship with staff from other agencies 
and understand the requirements that these agencies have and 
how adult education information can help these agencies address 
these needs. Furthermore, having a comparable management in- 
formation system and policies for confidentiality of data are essen- 
tial for any successful collaboration among agencies. 

Across-State Activities 

Another type of coordination is between or among state adult 
education offices. States in the same region or with similar delivery 
systems, populations of clients, or program improvement priorities 
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can develop joint products that address a common need, such as 
staff development, curriculum, or instructional resources. For this 
coordination to he successful, three conditions are necessary: 

• An idea “champion" must he available to lead the process; 

• There has to he a local commitment to pilot any processes or 
products that are developed; and 

• Expertise must he available to refine and develop the final 
product. 

For example, the states involved in the Northwest Regional Liter- 
acy Resource Center joined together in developing and pilot 
testing staff development modules and produced a Professional 
Development Series. They created processes for developing 
trainers and disseminating the training across the states. As a 
result of this coordination, the states can access a wider array of 
staff development modules than would have been possible without 
the joint effort. 

A second form of coordination is for stares to join together to share 
information, expertise, and existing resources in states, such as 
strategies for policy development, implementing performance stan- 
dards, and collecting quality data. For these types of activities, 
state representatives must commit to meet or communicate, 
experiment with processes, and report back to the group. 

State adult education staff also can transfer products, innovations, 
or effective practices between and among states, such as manage- 
ment information systems and staff development. In order for 
these activities to be successful, local technical support must be 
available, as well as resources for customizing the products and 
training and technical assistance for transferring the products and 
processes at a reasonable cost. All of these activities can help 
states leverage their limited resources and improve the quality of 
their adult education services. 



3il 



24 




A Final Word 



As state adult education policymakers move forward in stimulating 
continuous program improvement, determining more effective 
ways ot delivering services to adult learners, and collaborating with 
state agencies serving similar clients, they face a number of chal- 
lenges. The experience of state adult education policymakers has 
shown that there are a number of strategies that can be taken to 
address these challenges: 

• Develop statewide initiatives and pilot projects as strategies for 
focusing local programs’ attention on the need for continuous 
program improvement and experimenting with new programs 
and practices. 

• Encourage the adoption and adaptation of practices that have 
been shown to be effective as cost- and time-efficient ways of 
promoting program improvement. 

• Invest in training and technical assistance with all levels of pro- 
gram staff. 

• Set policies that are consistent and reinforce the statewide ini- 
tiatives and pilot projects that are being used to promote pro- 
gram change. 

• Consider intra- and interstate collaborations ns ways of lever- 
aging resources, creating high-quality products, and developing 
an infrastructure for serving clients more effectively. 

With every new challenge comes an opportunity. The field of 
adult education is at a critical point in having to demonstrate the 
effects of the services it provides, its impact on program partici- 
pants, and the ways in which its services support and relate to the 
country’s work force investment system. A tension underlies these 
requirements, with many practitioners concerned about the possi- 
ble new ways in which services are to be delivered to adults and 
their ability to continue to meet the needs of the learners as the 
learners define them. The mediation of this tension is a challenge 
for state and local adult education administrators, who must ad- 
i ess the new federal requirements while maintaining a robust and 
responsive service delivery sysrcin. In times like these, building a 
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community of teaching and learning is essential, and engaging in 
svsremic change is one strategy for coalescing support and creating 
a learning environment that enables individuals and organizations 
to grow personally and professionally. The discussion concerning 
approaches to systemic change presented in this document is offer' 
ed as one way to think about what might he undertaken and ac- 
complished with state and local leadership through a change 
initiative. 
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